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ABSTRACT^ ^ 

High School and Beyond is a national longitudinal > 
study of the 1980 classes of high school sophomores and seniors. It 
is part of a larger program of national longitudinal surveys 
commenced in 1972 by the National Cente^r for Education Statistics. 
The state of Washington participated directly in the study by 
expanding the number of schools selected (ji=50) for the national 
study to create valid state samples. Washington elected to survey the 
schools using National Opinion Resear,ch Center (NORC) materials and 
-^ocedures but with state personnel and resources. To assure the 
validity of comparisons with the national; results, NORC provided the 
data processing of the completed^ instruments . Because Washington was 
the only state to select this approach, the Washington High School 
and Beyond Study illustrates a uni<^ue example of a state's adaptation, 
of a large-scale national survey activity. The report describes the 
background of the study, the focus on school-level data, the 
development of interviewing skills, \the growing interest in the High 
School and Beyond data, the procedures used , concerns and problems 
encountered, and the benefits of interviewing as a follow-up 
technique. ( Author /PN) \ 
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PREFACE 



The Research on Evaluation Program is a Northwest Regional 
Educational Laboratory project of research, development, testing, 
cmd training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 
reports produced by program staff, visiting scholars, adjunct 
scholars, and project collaborators — -all members of a cooperative 
network of colleagues working on the development of new 
methodologies* 

Is interviewing using trained volunteers a cost-effective way of 
providing sij^plemental data to increase the local relevance of 
national survey information? Can small-scale studies be effec- 
tively piggy-backed onto larger national efforts to increase 
their local utility? Based on the project described here, the 
answer to both these questions is yes. This project, mounted by 
the testing and evaluation unit of the Washington state 
department of education, illustrates a cost-effective way to 
gather siq>plemental qualitative and quantitative information of 
increased local utility by properly augmenting a national study — 
illustrating an important design option- iti^'this period of 
decreasing evaluation resources. 

Nick L. Smith, Editor 
Paper and Report Series 
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INTRODUCTION 

In 1980 the Washington State Superintendent of Public 
Instruction's Testing .and Evaluation Section participated in the 
High School and Beyond Longitudinal Study funded by the National 
Center for Education Statistics and directed by James Coleman of 
the National Opinion Research Center, To augment the 15 Washington 
schools selected in the national sample an additional 50 were selectee 
by Coleman to serve as the Washington State sample of 65 schools. 
Tenth and twelveth grade students in these schools were randomly 
selected to participate and a total of 3,645 responded to the 
study's questionnaires and tests. The data collection procedures 
called for two-person teams of professional educators to visit 
each of the schools at least twice: first to orient the school 
coordinators and students, and second to administer the questionnaire 
and tests and to pick up the principal's School Questionnaire, 
The report that follows is primarily concerned with the strategies 
employed for collecting information on the School Questionnaire, 

The 65 item School Questionnaire, which was either mailed 
to the principal or delivered during the orientation visit — 
allowing ample time for completion — is the source of much 
descriptive information about the school and the perceptions of 
the principal. The intent of the School Questionnaire, beyond 
providing basic demographic data, was to gain useful information 
for conducting school effectiveness research, A preliminary editing 
of selected items, quickly demonstrated that the power of the data 
was greatly limited by confusing questions, careless responses, 
and frequent omissions. 

The purpose of the contract between the Washington State 
Superintendent of Public Instruction and the Northwest Regional 
Educational Laboratory was to use interviewing techniques in the 
sample schools to augment the School Questionnaire data, analyze 
the interview data, and to report on the utility of the interview 

This report was developed under a contractual agreement with the 
Northwest Regional Educational Laboratory, The information 
presented, however, does not necessarily reflect the views of that 
agency. 



procedures. The NWREL's underlying concern was methodological — 
would a relatively small effort using interview techniques expand 
the type of information available and increase the state and local 
utility and relevance of the survey data. If the effort proved 
usefixl, it .would suggest 1) that local, on-site interviewing can 
be accomplished using trained volunteers, 2) that interview data 
can supplement survey data, 3) that interviewing can be a low cost 
strategy for increasing the utility of local data gathered for 
national research purposes, and 4) that ''piggy-backing" a small 
scale study onto a larger national effort is a cost effective 
procedure for states and local school districts to consider. 

To achieve these purposes the original School Questionnaire 
results were analyzed. Questions with high rates of no responses 
or improbable responses, along with those of special interest to 
the interview team, were selected as part of the interview protocol — 
a^roximately 20 percent of the original School Questionnaire items 
were included. In addition, an addendum of 25 new items was developed, 
pilot tested, and used in the interviewing process. A grpup composed 
of nine school administrators and four members of the state's Testing 
and Evaluation Section was trained by the section's director. Each 
of the 65 schools was assigned to an interviewer, and on-site 
interviews were conducted primarily during May and June, 1981. 

The report that follows describes the background of the study, 
the focus on school level data, the development of interviewing 
skills, the growing interest in the High School and Beyond data, 
the procedures used, 'concerns and problems encountered, and the 
benefits of interviewing as a follow-up technique. 



BACKGROUND 

High School and Beyond is a national longitudinal study of 
the 1980 classes of high school sophomores and seniors. It is 
part of a larger program of national longitudinal surveys commenced 
in 1972 by the National Center for Education Statistics, At this 
time the long-range plan for the 1980 sophomores and seniors builds 
on the base year data with follow-ups scheduled every two years 
throughout the decade , 

Prior to initiating the 1980 survey, the National Opinion 
Research Center (NORC) , following recommendations from the Council 
of Chief ^ State School Officers, offered each state the opportunity 
to participate directly in the study by expanding the numbejr of 
schools selected for the national study to create valid state 
samples. The Department of Defense overseas schools and six 
states elected to do this — California, Illinois, New York, Ohio, 
Pennsylvania, and Washington, Because of their size, five of the 
states needed to add only a few schools to complete state samples, 
These states elected to have NORC collect and analyze the data 
from their additional schools. Due to its smaller population, 
only 15 Washington schools were selected in the national sample, 
and it was determined that a total of 65 schools would be required 
for an adequate state sample, 

- Washington's interest in the potential information was high, 
but the cost of contracting with NORC to collect data from 50 
additional schools was prohibitive. Therefore, Washington elected 
to survey the 50 schools using NORC materials and procedures but 
with state personnel and resources. To assure the validity of 
comparisons with the national results NORC provided the data 
processing of the completed instruments. Because Washington was 
the only state to select this approach, the Washington High School 
and Beyond Study illustrates a unique example of a state's adaptation^ 
of a large scale national survey activity. 
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Participation in the fetudy was attractive to Washington for 
two reasons- First, the state had little information about high 
school students that was current and representative. Second, the 
Washington High School and Beyond survey would provide a rich 
source of information for state and local planning and decision 
making. 

At the beginning of Washington's participation in High School 
and Beyond , a close working relationship was established between 
the State Superintendent of Public Instruction's office and the 
Association of Washington School Principals, The Principal's 
Association assisted by reviewing the survey materials, by helping 
to establish initial acceptance of and support for the survey, 
and by contracting with ^ some of the field work personnel. In late 
1980 the Bureau of School Service and Research, University of 
Washington, was added to the analysis team as technical advisor 
and computer consultant 



FOCUS SCHOOL LEVEL DATA 

As outlined in the INTRODUCTION this research effort focused 
on the use of interviews to add to and improve the quality of the 
data collected by the School Questionnaire survey. The responses 
to the School Questionnaire are a significant part of the High 
School and Beyond Study because they generate a school level basis 
for comparing student *s perceptions of their backgrounds, schooling 
and aspirations, as well as, their achievements. The original 
questionnaires completed by principals during the spring of 1980 
provided information d^n a variety of school variables, many of 
which are related to t|he school effectiveness literature. For 
example, principals we^re asked for the numbers of students and 
grades served by the school, the number of books in the library, 
the number of days in the school year and length and number of 
periods in the school day. The principals also reported the type 
of school, programs available, classes taught, and percent of 

students enrolled in selected areas. The principals estimated 

f 

the ayerage daily attendance, percent of dropouts, and percent 
of students and faculty in minority groups. School organization, 
grading practices, operating procedures, school facilities, finances, 
and staff characteristics were described. The principals also 
responded to questions related to special student populations and 
to problems in their schools. Additionally, in 1980 when the field 
work was conducted the State Superintendent's office attempted 
to gather information regarding school organization through eight 
observation and interview questions on a School Organizational 
Survey. 

Several problems became apparent after a preliminary 
review of the School Questionnaire responses. First, too many 
answers were omitted. Although th'e^e was no question that demanded 
knowledge beyond that held by the principal or selected members 
of the school staff, many questions were skipped — few questionnaires 
were complete. 
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Second, closer inspectipn of item responses showed that several 
of the questions were frequently, misread resulting in responses 
that were in the category of "possible but not probable." It was 
equally clear that the field workers had not received enough 
training to standardize the observational techniques relating^, to 
school organization. As a result it was difficult to distinguish 
among the schools based on the brief interviews and observations, 
^ence the reliability of data was low. 

Although the quality of the results of the School Questionnaire 
and organizational survey was disappointing in itself, there was 
additionally, a keen interest on the part of several Washington 
researchers, to seek more information regarding educational practice^ 
' in the sample schools. Tbls interest focused on three general 
areas: staff evaluation, school organization, and decision-making 
practices related to the instructional program. The plan for using 
interviews to gain more and better data began to take shape. 
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DEVELOPMENT OF INTERVIEWING SKILLS 

Evaluation specialists, in a variety of settings, are frequently 
asked to provide data which is descriptive of a particular program 
or setting and yet more complete than a simple inventory of "things" 
or responses to a written questionnaire or survey. Face-to-face 
interviewing offers an opportunity to enrich descriptions of 
programs while at the same time improving both the response rate 
and the validity of the respgnses, 

The purpose of intervi^ing, in the i)resent project, was to 
obtain a description of certain characteristics of selected high 
schools in Washington, Although interviewing can be quite informal, 
effective interviewing requires some minimum skills and practice 
in at least the following areas: 1) language literacy, 2) reasonable 
sensitivity to the persons being interviewed and their environments 
(in this case high schools) , and 3) ability to communicate reliably 
the important dimensions of the responses obtained 1,'n the interviews. 
Good interviewing should result in a cle^r sense of what happens 
in a particular setting (i.e. high school) or how the interviewee 
(i.e. principal) thinks or feels about a particular issue related 
to the setting. 

An interest in interviewing as an evaluation technique follows 
from the belief that it is important to know, in a more naturalistic 
way, about the context and processes operating in a program or 
setting. While the results of interviewing can provide complimentary 
data about a program, they do not replace other evaluative techniques, \^ 
They do, however, offer additional, useful perspectives. 

Further, data gathered by the interview technique have important 
virtues for policy making^,- Frequently, in making decisions in 
areas where competing viewpoin^ts exist* -implementation of a policy 
is enhanced when decision makers know both the "facts" of the 
situation and how important participants ^el about the facts, 
Interviewirig offers an opportunity to collec^botlr-tbe^f ac ts and 
the perceptions of the facts. 
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Recognizing that effective interviewing requires observation 
and personal interaction skills, members of the State Superintendent^ 
staff participated in two training sessions focusing on the skills 
and techniques of interviewing. These training sessions included 
substantial opportunities for practice and made use of video taping 
to provide feedback to the trainees and provide opportunities for 
participants and trainers to critique interviewing skills and 
styles. 

The sessions were each two days in length and were conducted 
on the Central Washington University campus in Ellensburg during 
October, 1979, and December, 1980, This setting afforded easy 
access to both video and recording equipment and individual 
interviewing rooms with one-way observation windows, 

The focus of the sessions was on the development and use of 
Interview protocols and on techniques for eliciting cooperative 
responses during interview sessions. Dr, Terry Denny was the 
seminar trainer' for these sessions, Dr, Denny is the Assistant 
Dean for Graduate Programs, College of Education, University of 
Illinois, and a recognized authority ±n the field of interviewing 
methodology. 
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GROWING INTEREST, IN USE OF HIGH SCHOOL AND BEYOND DATA 

As the amount of information generated by national studies 
of secondary education grows, there seems to be a corresponding 
growth in the -interest in using this information for additional 
research purposes and policy formation. The impact of the Washington 
High School and Beyond data clearly fits this pattern. During 
the original 1980 data collection, two school^administrators, one 
a high school principal and the other a director of secondary 
education, asked to participate in the field work in return for 
access to the data for dissertation purposes.* Their offer of 
assistance was accepted, they were trained, and served together 
as one of the field teams- The success of their involvement 
encouraged other graduate students in educational administration 
to offer assistance in subsequent endeavors. 

The fact that James Coleman is the chief investigator of the 
national phase of the study generates national, even international 
interest. His name is well known, his work is often controversial, 
and the news coverage of his recent efforts and findings lends 
the study a certain mystique causing both educators and the public 
to pay attention. 

A meeting titled the Six State Seminar held at the 
University of Chicago in early May, 1981, gave a research team 
from Washington State a chance to meet and exchange ideas with 
Coleman and members of his National Opinion Research Center staff, 
representatives of the National Center for Education Statistics, 
and researchers and evaluators from California, Illinois^ New York, 
Ohio and Pennsylvania. The Washington contingent, consisting of 
university scholars and LEA and SEA researchers, played a central 
role in the meeting because Washington had cond.ucted most of its 
own field work and the preliminary analysis of the Washington 
results was well under way. The intellectual reinforcement 
received stimulated the planning and implementation of the next 
steps. 



*At this time one of these educators had completed the doctorate 
and the other played a strategic role in the interviewing project 
being described. 
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Specifically the experience in Chicago reinforced the desire 
to follow-up on the preliminary discussions with the Northwest 
Regional Educational Laboratory to correct and supplement survey 
data through the use of local interviews. There was no doubt that 
the responses to the School Questionnaire were unreliable in many 
instances and that steps should be taken to improve the situation. 
The final decision was made to interview the principals of the 
schools in the sample. 
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ORGANIZING FOR INTERVIEWING 

After the return from Chicago, the call went out for field 
work volunteers. Two doctoral students at the University of 
Washington, both school administrators in secondary schools, 
provided special leadership in the project. One of these had been 
involved in the 1980 data collection; both were interested in 
augmenting the School Questionnaire to gain additional information 
for their doctoral research. With their assistance a strong cadre 
of educsctors was enlisted to supplement the efforts of the state 
education department's evaluation staff. All of the volunteers 
were school administrators, most were enrolled in doctoral studies 
with four universities represented. 

The cooperation of their districts in providing released 
time for the training and interviewing in the schools was 
remarkable. The answer to **Why the willingness?" was based on 
the district level s(dmiuistrators sincere interest in "doing their 
share" to advance educational research. The importance of Coleman 
involvement, the state's decision to visit the schools and collect 
data first-hand, and the excitement of a special study within a 
study as contracted by the Northwest Laboratory should not be 
overlooked as positive factors enlisting support. 

In order to prepare for the training of the interviewers 
three activities had to be undertaken. First, a decision was 
required regarding the items from the School Questionnaire to be 
included in the interview schedule. Second, the additional resear 
interests needed to be formed into questions as an addendum' to 
the School Questionnaire. Third, the interview protocol had to 
be developed and tried out. 

The computer at the University of Washington was programmjed 
to print the responses to each of the 65^uestions to determine 
the survey items that resulted in the least acceptable data, that 
is, to locate omissions and improbable responses. Twelve original 
School Questionnaire items were identified for the interviews, 
three were modified and included in the addendum. The identified 



items typically involved numbers or had directions that were 
confusing, complex or ambiguous • , ^ 

Additional items exploring staff evaluation, school organization, 
and instructional decision making were discussed, developed and 
organized into an addendumtto the School Questionnaire, As mentioned 
previously this effort was implemented with the help of two University 
of Washington doctoral students. 

An interview protocol was developed, tested and revised. 
Efforts were made to ensure that each interview would take less 
than 60 minutes, with a target of 45 minutes- The concern for 
reliability among interviewers led to the development of procedures 
for standardizing introductions, the asking of questions, and the 
recording of responses. 

In general each interviewer was assigned four or five schoo;Ls. 
Final assignments were made during the training session so that 
both proximity to the assigned schools and acquaintance with the 
principals could be incorporated in the final selections to save 
travel funds and enhance the on-site cooperation. The assignment 
process caused only one problem — the volunteers were so eager 
that several wanted to visit more schools. 

The cost of the interviewing project was low because no money 
was required for salaries or released time. The project's budget 
essentially covered travel expenses for the field workers (although 
for short trips, interviewers often did not seek reimbursement) 
and some of the costs for computer storage, tabulation and analysis 
of data. 
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CONCERNS AND PRQBLEMS ENCOUNTERED 

The use of follow-up interviews as a device for completing 
and expandiug questionnaire data requires that attention be given 
to the training of interviewers. In the present situation it was 
necessary to review basic interviewing procedures to help assure 
that reliable , data would be collected. Therefore a training and 
planning session, requiring about two hours of group effort, was 
conducted and proved to be extremely worthwhile. In this session, 
procedures for the collection of unbiased and reliable data were 
reviewed, along with general introductory comments regarding reasons 
for the research and the confidentiality assurances to be given .\ 

An interview protocol was developed which reconstructed the 
scene of the data collection from the previous school year. During 
the interview itself, frequent reminders of the previous data 
collection efforts were necessary. Additionally, the interviewers 
had to be prepared to recognize the usual indicators of "memory 
gaps" so that they could probe for more accurate data. In such 
cases, the respondent was reminded that it was important to provide 
information that was as accurate as possible. By simply stating 
that time does tend to "soften and smooth out" incidents in one's 
memory, the interviewers refocused the respondents' desire to 
provide reliable data for the study. During the interview 
training session, the interviewers were also alerted to the 
problem of differentiating between precise numbers and wild 
estimates. By recognizing the "estiiiijate cues" used by the 
respondent, the interviewer could theii pursue a line of questioning 
designed to yield more precise information. 

In using the interview to follow-up on questionnaire data, 
occasionally, some effort was required to locate the original 
respondent who had changed jobs since the original completion of 
the questionnaire. This necessitated additional time and travel 
costs but did not prove to be an insurmountable problem. 
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BENEFITS OF INTERVIEWING 

A number of specific benefits accrued from the interviews 
with school principals — some anticipated, some serendipitous. 
The most obvious anticipated outcome was the correction and/or 
verification of responses to selected items on the original suz^vey. 
Editing of the original responses had raised questions about the 
accuracy of some of the information. The interviews provided the 
opportunity to make certain that principals understood what th&y 
were being asked before responding. The interviews also resulted 
in the collection of responses that had been omitted on the original 
surveys. 

Since the interview protocols focused only on selected items 
from the original School Questionnaire survey, there was an 
opportunity to explore additional areas of interest to the State 
Superintendent and the teams of interviewers. Considerable interview 
time was focused on the principal's involvement in teacher evaluation 
and with the principal's responsibilities for and involvement In 
curriculum planning and evaluation. In both of these areas, the 
interviewing process was helpful in that appropriate probes or 
questions by the interviewer resulted in more and qualitatively 
better Information than what could have been gathered by a written 
survey instrument alone. 

An important serendipitous benefit of the interviewing was 
the very positive interaction between the field workers and the, 
principals. Certainly, all interviewers were cordially received, 
but following the interviews, many principals expressed a genudlne 
enthusiasm for the opportunity to discuss some important topics 
at some length with an interested and empathic listener. And since 
most of the interviewers were themselves practicii^tg administrators, 
the interviews provided them an opportunity for r^few insights and 
perspectives about the principal's role and responsibilities. 
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SUMMARY 

Although minor problems arose in using interviewing techniques 
to augment the survey data, the face-to-face strategies generally 
worked well. The relatively small effort did result in the 
generation of more accurate and additional information for 
research and policy analysis at the state and local levels- 

The study clearly demonstrated that local school administrators 
and graduate student volunteers can be trained to conduct productive 
on-site interviews. The training in this instance focused on a 
major concern for reliability. The interview team reviewed the 
items to be included in the interviews and together agreed on the 
intent of the question, data desired, range of probable responses, 
problem areas, and special probes or procedures required. The 
training was adequate for the task, but in retrospect a short 
session on the "theory" of interviewing and a written summary of 
the group's agreement on the treatment of each 'item would have 
been helpful* 

The interviews did improve the quality and completeness of 
data generated by the original paper and pencil survey- Verification 
probes were used to gain answers to questions frequently omitted 
on the survey and to double check the accuracy and appropriateness 
of responses to complex questions calling for descriptive detail- 
Interviewing can be a low cost strategy for collecting data- 
By making the data available for graduate student researchers, 
by training and working with volunteers who have an interest using 
the data or gaining new skills, by dividihg the total effort into 
manageable parts, by assigning locations with care — interviews 
were conducted efficiently and effectively in 65 high schools 
scattered across the state. Perhaps the greatest problem 
encountered wzis the time consuming chore of actually scheduling 
interviews. 

The reduced costs of materials and technical assistance, the 
resulting mass of research and policy data, and the excitement 
of involvement strongly reinforce the advantages of state and local 
agencies coordinating with, "piggy-backing" on, major national 
investigations. 
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